{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NLP Pipeline with spaCy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[spaCy](https://spacy.io/) is a widely used python library with a comprehensive feature set for fast text processing in multiple languages. \n",
    "\n",
    "The usage of the tokenization and annotation engines requires the installation of language models. The features we will use in this chapter only require the small models, the larger models also include word vectors that we will cover in chapter 15."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![spaCy](assets/spacy.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:28.422236Z",
     "start_time": "2018-11-22T18:23:27.574043Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import sys\n",
    "import tarfile\n",
    "from pathlib import Path\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import spacy\n",
    "from spacy import displacy\n",
    "import textacy\n",
    "\n",
    "from IPython.display import SVG, display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:28.425592Z",
     "start_time": "2018-11-22T18:23:28.423600Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "pd.set_option('float_format', '{:,.2f}'.format)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### SpaCy Language Model Installation\n",
    "\n",
    "In addition to the `spaCy` library, we need [language models](https://spacy.io/usage/models)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### English"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:29.715775Z",
     "start_time": "2018-11-22T18:23:28.426957Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: en_core_web_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz#egg=en_core_web_sm==2.0.0 in /home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages (2.0.0)\n",
      "\n",
      "\u001b[93m    Linking successful\u001b[0m\n",
      "    /home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/en_core_web_sm\n",
      "    -->\n",
      "    /home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/spacy/data/en_core_web_sm\n",
      "\n",
      "    You can now load the model via spacy.load('en_core_web_sm')\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python -m spacy download en_core_web_sm\n",
    "\n",
    "# more comprehensive models:\n",
    "# {sys.executable} -m spacy download en_core_web_md\n",
    "# {sys.executable} -m spacy download en_core_web_lg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Spanish"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[Spanish language models](https://spacy.io/models/es#es_core_news_sm) trained on [AnCora Corpus](http://clic.ub.edu/corpus/) and [WikiNER](http://schwa.org/projects/resources/wiki/Wikiner)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:30.703639Z",
     "start_time": "2018-11-22T18:23:29.717730Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: es_core_news_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/es_core_news_sm-2.0.0/es_core_news_sm-2.0.0.tar.gz#egg=es_core_news_sm==2.0.0 in /home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages (2.0.0)\n",
      "\n",
      "\u001b[93m    Linking successful\u001b[0m\n",
      "    /home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/es_core_news_sm\n",
      "    -->\n",
      "    /home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/spacy/data/es_core_news_sm\n",
      "\n",
      "    You can now load the model via spacy.load('es_core_news_sm')\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python -m spacy download es_core_news_sm\n",
    "\n",
    "# more comprehensive model:\n",
    "# {sys.executable} -m spacy download es_core_news_md"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create shortcut names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:31.456755Z",
     "start_time": "2018-11-22T18:23:30.705604Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[93m    Linking successful\u001b[0m\n",
      "    /home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/en_core_web_sm\n",
      "    -->\n",
      "    /home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/spacy/data/en\n",
      "\n",
      "    You can now load the model via spacy.load('en')\n",
      "\n",
      "\n",
      "\u001b[93m    Linking successful\u001b[0m\n",
      "    /home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/es_core_news_sm\n",
      "    -->\n",
      "    /home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/spacy/data/es\n",
      "\n",
      "    You can now load the model via spacy.load('es')\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python -m spacy link en_core_web_sm en --force;\n",
    "python -m spacy link es_core_news_sm es --force;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Validate Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:32.225454Z",
     "start_time": "2018-11-22T18:23:31.458167Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "\u001b[93m    Installed models (spaCy v2.0.16)\u001b[0m\r\n",
      "    /home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/spacy\r\n",
      "\r\n",
      "    TYPE        NAME                  MODEL                 VERSION                                   \r\n",
      "    package     es-core-news-sm       es_core_news_sm       \u001b[38;5;2m2.0.0\u001b[0m    \u001b[38;5;2m✔\u001b[0m      \r\n",
      "    package     en-core-web-sm        en_core_web_sm        \u001b[38;5;2m2.0.0\u001b[0m    \u001b[38;5;2m✔\u001b[0m      \r\n",
      "    link        en_core_web_sm        en_core_web_sm        \u001b[38;5;2m2.0.0\u001b[0m    \u001b[38;5;2m✔\u001b[0m      \r\n",
      "    link        en                    en_core_web_sm        \u001b[38;5;2m2.0.0\u001b[0m    \u001b[38;5;2m✔\u001b[0m      \r\n",
      "    link        es                    es_core_news_sm       \u001b[38;5;2m2.0.0\u001b[0m    \u001b[38;5;2m✔\u001b[0m      \r\n",
      "    link        es_core_news_sm       es_core_news_sm       \u001b[38;5;2m2.0.0\u001b[0m    \u001b[38;5;2m✔\u001b[0m      \r\n"
     ]
    }
   ],
   "source": [
    "# validate installation\n",
    "!{sys.executable} -m spacy validate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Get Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download and unzip into the folder `bbc` in your main data directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_DIR = Path('../data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:22:17.894231Z",
     "start_time": "2018-11-22T18:22:17.889858Z"
    }
   },
   "source": [
    "- [BBC Articles](http://mlg.ucd.ie/datasets/bbc.html), use raw text files (download [link](http://mlg.ucd.ie/files/datasets/bbcsport-fulltext.zip)\n",
    "- [TED2013](http://opus.nlpl.eu/TED2013.php), a parallel corpus of TED talk subtitles in 15 langugages (sample provided)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## SpaCy Pipeline & Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### The Processing Pipeline\n",
    "\n",
    "When you call a spaCy model on a text, spaCy \n",
    "\n",
    "1) tokenizes the text to produce a `Doc` object. \n",
    "\n",
    "2) passes the `Doc` object through the processing pipeline that may be customized, and for the default models consists of\n",
    "- a tagger, \n",
    "- a parser and \n",
    "- an entity recognizer. \n",
    "\n",
    "Each pipeline component returns the processed Doc, which is then passed on to the next component.\n",
    "\n",
    "![Architecture](assets/pipeline.svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Key Data Structures\n",
    "\n",
    "The central data structures in spaCy are the **Doc** and the **Vocab**. Text annotations are also designed to allow a single source of truth:\n",
    "\n",
    "- The **`Doc`** object owns the sequence of tokens and all their annotations. `Span` and `Token` are views that point into it. It is constructed by the `Tokenizer`, and then modified in place by the components of the pipeline. \n",
    "- The **`Vocab`** object owns a set of look-up tables that make common information available across documents. \n",
    "- The **`Language`** object coordinates these components. It takes raw text and sends it through the pipeline, returning an annotated document. It also orchestrates training and serialization.\n",
    "\n",
    "![Architecture](assets/spaCy-architecture.svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## SpaCy in Action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Create & Explore the Language Object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once installed and linked, we can instantiate a spaCy language model and then call it on a document. As a result, spaCy produces a Doc object that tokenizes the text and processes it according to configurable pipeline components that by default consist of a tagger, a parser, and a named-entity recognizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:32.685811Z",
     "start_time": "2018-11-22T18:23:32.231489Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "nlp = spacy.load('en') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:32.690366Z",
     "start_time": "2018-11-22T18:23:32.686964Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "spacy.lang.en.English"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(nlp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:32.698471Z",
     "start_time": "2018-11-22T18:23:32.691679Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'en'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp.lang"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:32.710213Z",
     "start_time": "2018-11-22T18:23:32.699643Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    \u001b[93mInfo about model en\u001b[0m\n",
      "\n",
      "    lang               en             \n",
      "    pipeline           ['tagger', 'parser', 'ner']\n",
      "    accuracy           {'token_acc': 99.8698372794, 'ents_p': 84.9664503965, 'ents_r': 85.6312524451, 'uas': 91.7237657538, 'tags_acc': 97.0403350292, 'ents_f': 85.2975560875, 'las': 89.800872413}\n",
      "    name               core_web_sm    \n",
      "    license            CC BY-SA 3.0   \n",
      "    author             Explosion AI   \n",
      "    url                https://explosion.ai\n",
      "    vectors            {'keys': 0, 'width': 0, 'vectors': 0}\n",
      "    sources            ['OntoNotes 5', 'Common Crawl']\n",
      "    version            2.0.0          \n",
      "    spacy_version      >=2.0.0a18     \n",
      "    parent_package     spacy          \n",
      "    speed              {'gpu': None, 'nwords': 291344, 'cpu': 5122.3040471407}\n",
      "    email              contact@explosion.ai\n",
      "    description        English multi-task CNN trained on OntoNotes, with GloVe vectors trained on Common Crawl. Assigns word vectors, context-specific token vectors, POS tags, dependency parse and named entities.\n",
      "    link               /home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/spacy/data/en\n",
      "    source             /home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/en_core_web_sm\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'lang': 'en',\n",
       " 'pipeline': ['tagger', 'parser', 'ner'],\n",
       " 'accuracy': {'token_acc': 99.8698372794,\n",
       "  'ents_p': 84.9664503965,\n",
       "  'ents_r': 85.6312524451,\n",
       "  'uas': 91.7237657538,\n",
       "  'tags_acc': 97.0403350292,\n",
       "  'ents_f': 85.2975560875,\n",
       "  'las': 89.800872413},\n",
       " 'name': 'core_web_sm',\n",
       " 'license': 'CC BY-SA 3.0',\n",
       " 'author': 'Explosion AI',\n",
       " 'url': 'https://explosion.ai',\n",
       " 'vectors': {'keys': 0, 'width': 0, 'vectors': 0},\n",
       " 'sources': ['OntoNotes 5', 'Common Crawl'],\n",
       " 'version': '2.0.0',\n",
       " 'spacy_version': '>=2.0.0a18',\n",
       " 'parent_package': 'spacy',\n",
       " 'speed': {'gpu': None, 'nwords': 291344, 'cpu': 5122.3040471407},\n",
       " 'email': 'contact@explosion.ai',\n",
       " 'description': 'English multi-task CNN trained on OntoNotes, with GloVe vectors trained on Common Crawl. Assigns word vectors, context-specific token vectors, POS tags, dependency parse and named entities.',\n",
       " 'link': '/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/spacy/data/en',\n",
       " 'source': '/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/en_core_web_sm'}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spacy.info('en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:32.715948Z",
     "start_time": "2018-11-22T18:23:32.711601Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def get_attributes(f):\n",
    "    print([a for a in dir(f) if not a.startswith('_')], end=' ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:32.724335Z",
     "start_time": "2018-11-22T18:23:32.717007Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Defaults', 'add_pipe', 'begin_training', 'create_pipe', 'disable_pipes', 'entity', 'evaluate', 'factories', 'from_bytes', 'from_disk', 'get_pipe', 'has_pipe', 'lang', 'make_doc', 'matcher', 'max_length', 'meta', 'parser', 'path', 'pipe', 'pipe_names', 'pipeline', 'preprocess_gold', 'remove_pipe', 'rename_pipe', 'replace_pipe', 'tagger', 'tensorizer', 'to_bytes', 'to_disk', 'tokenizer', 'update', 'use_params', 'vocab'] "
     ]
    }
   ],
   "source": [
    "get_attributes(nlp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Explore the Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s illustrate the pipeline using a simple sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:32.756807Z",
     "start_time": "2018-11-22T18:23:32.725598Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "sample_text = 'Apple is looking at buying U.K. startup for $1 billion'\n",
    "doc = nlp(sample_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:32.761010Z",
     "start_time": "2018-11-22T18:23:32.758505Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['cats', 'char_span', 'count_by', 'doc', 'ents', 'extend_tensor', 'from_array', 'from_bytes', 'from_disk', 'get_extension', 'get_lca_matrix', 'has_extension', 'has_vector', 'is_parsed', 'is_sentenced', 'is_tagged', 'mem', 'merge', 'noun_chunks', 'noun_chunks_iterator', 'print_tree', 'remove_extension', 'retokenize', 'sentiment', 'sents', 'set_extension', 'similarity', 'tensor', 'text', 'text_with_ws', 'to_array', 'to_bytes', 'to_disk', 'user_data', 'user_hooks', 'user_span_hooks', 'user_token_hooks', 'vector', 'vector_norm', 'vocab'] "
     ]
    }
   ],
   "source": [
    "get_attributes(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:32.774134Z",
     "start_time": "2018-11-22T18:23:32.767812Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc.is_parsed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:32.779002Z",
     "start_time": "2018-11-22T18:23:32.775876Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc.is_sentenced"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:32.788762Z",
     "start_time": "2018-11-22T18:23:32.780469Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc.is_tagged"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:32.802336Z",
     "start_time": "2018-11-22T18:23:32.794045Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Apple is looking at buying U.K. startup for $1 billion'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:32.807584Z",
     "start_time": "2018-11-22T18:23:32.804542Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['add_flag', 'cfg', 'data_dir', 'from_bytes', 'from_disk', 'get_vector', 'has_vector', 'lang', 'length', 'lex_attr_getters', 'lexemes_from_bytes', 'lexemes_to_bytes', 'morphology', 'prune_vectors', 'reset_vectors', 'set_vector', 'strings', 'to_bytes', 'to_disk', 'vectors', 'vectors_length'] "
     ]
    }
   ],
   "source": [
    "get_attributes(doc.vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:32.818729Z",
     "start_time": "2018-11-22T18:23:32.809437Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "57852"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc.vocab.length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Explore `Token` annotations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The parsed document content is iterable and each element has numerous attributes produced by the processing pipeline. The below sample illustrates how to access the following attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:23:32.827692Z",
     "start_time": "2018-11-22T18:23:32.820215Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       Apple\n",
       "1          is\n",
       "2     looking\n",
       "3          at\n",
       "4      buying\n",
       "5        U.K.\n",
       "6     startup\n",
       "7         for\n",
       "8           $\n",
       "9           1\n",
       "10    billion\n",
       "dtype: object"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series([token.text for token in doc])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:24:11.570806Z",
     "start_time": "2018-11-22T18:24:11.549908Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>lemma</th>\n",
       "      <th>pos</th>\n",
       "      <th>tag</th>\n",
       "      <th>dep</th>\n",
       "      <th>shape</th>\n",
       "      <th>is_alpha</th>\n",
       "      <th>is_stop</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Apple</td>\n",
       "      <td>apple</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>nsubj</td>\n",
       "      <td>Xxxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>is</td>\n",
       "      <td>be</td>\n",
       "      <td>VERB</td>\n",
       "      <td>VBZ</td>\n",
       "      <td>aux</td>\n",
       "      <td>xx</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>looking</td>\n",
       "      <td>look</td>\n",
       "      <td>VERB</td>\n",
       "      <td>VBG</td>\n",
       "      <td>ROOT</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>at</td>\n",
       "      <td>at</td>\n",
       "      <td>ADP</td>\n",
       "      <td>IN</td>\n",
       "      <td>prep</td>\n",
       "      <td>xx</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>buying</td>\n",
       "      <td>buy</td>\n",
       "      <td>VERB</td>\n",
       "      <td>VBG</td>\n",
       "      <td>pcomp</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>U.K.</td>\n",
       "      <td>u.k.</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>NNP</td>\n",
       "      <td>compound</td>\n",
       "      <td>X.X.</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>startup</td>\n",
       "      <td>startup</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>NN</td>\n",
       "      <td>dobj</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>for</td>\n",
       "      <td>for</td>\n",
       "      <td>ADP</td>\n",
       "      <td>IN</td>\n",
       "      <td>prep</td>\n",
       "      <td>xxx</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>$</td>\n",
       "      <td>$</td>\n",
       "      <td>SYM</td>\n",
       "      <td>$</td>\n",
       "      <td>quantmod</td>\n",
       "      <td>$</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NUM</td>\n",
       "      <td>CD</td>\n",
       "      <td>compound</td>\n",
       "      <td>d</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>billion</td>\n",
       "      <td>billion</td>\n",
       "      <td>NUM</td>\n",
       "      <td>CD</td>\n",
       "      <td>pobj</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       text    lemma    pos  tag       dep  shape  is_alpha  is_stop\n",
       "0     Apple    apple  PROPN  NNP     nsubj  Xxxxx      True    False\n",
       "1        is       be   VERB  VBZ       aux     xx      True     True\n",
       "2   looking     look   VERB  VBG      ROOT   xxxx      True    False\n",
       "3        at       at    ADP   IN      prep     xx      True     True\n",
       "4    buying      buy   VERB  VBG     pcomp   xxxx      True    False\n",
       "5      U.K.     u.k.  PROPN  NNP  compound   X.X.     False    False\n",
       "6   startup  startup   NOUN   NN      dobj   xxxx      True    False\n",
       "7       for      for    ADP   IN      prep    xxx      True     True\n",
       "8         $        $    SYM    $  quantmod      $     False    False\n",
       "9         1        1    NUM   CD  compound      d     False    False\n",
       "10  billion  billion    NUM   CD      pobj   xxxx      True    False"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame([[t.text, t.lemma_, t.pos_, t.tag_, t.dep_, t.shape_, t.is_alpha, t.is_stop]\n",
    "              for t in doc],\n",
    "             columns=['text', 'lemma', 'pos', 'tag', 'dep', 'shape', 'is_alpha', 'is_stop'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Visualize POS Dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualize the syntactic dependency in a browser or notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "options = {'compact': True, 'bg': '#09a3d5',\n",
    "           'color': 'white', 'font': 'Source Sans Pro', 'notebook': True}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:24:23.879627Z",
     "start_time": "2018-11-22T18:24:11.572586Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[93m    Serving on port 5000...\u001b[0m\n",
      "    Using the 'dep' visualizer\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "127.0.0.1 - - [20/Apr/2019 18:24:26] \"GET / HTTP/1.1\" 200 8357\n",
      "127.0.0.1 - - [20/Apr/2019 18:24:26] \"GET /favicon.ico HTTP/1.1\" 200 8357\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    Shutting down server on port 5000.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "displacy.serve(doc, style='dep', options=options)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:24:23.883977Z",
     "start_time": "2018-11-22T18:24:23.880703Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" id=\"0\" class=\"displacy\" width=\"1700\" height=\"362.0\" style=\"max-width: none; height: 362.0px; color: white; background: #09a3d5; font-family: Source Sans Pro\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"272.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">Apple</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"272.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"200\">is</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"200\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"272.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"350\">looking</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"350\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"272.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"500\">at</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"500\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"272.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"650\">buying</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"650\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"272.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"800\">U.K.</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"800\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"272.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"950\">startup</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"950\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"272.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1100\">for</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1100\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"272.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1250\">$</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1250\">SYM</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"272.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1400\">1</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1400\">NUM</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"272.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1550\">billion</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1550\">NUM</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-0\" stroke-width=\"2px\" d=\"M62,227.0 62,177.0 347.0,177.0 347.0,227.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-0\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M62,229.0 L58,221.0 66,221.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-1\" stroke-width=\"2px\" d=\"M212,227.0 212,202.0 344.0,202.0 344.0,227.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-1\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">aux</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M212,229.0 L208,221.0 216,221.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-2\" stroke-width=\"2px\" d=\"M362,227.0 362,202.0 494.0,202.0 494.0,227.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-2\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">prep</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M494.0,229.0 L498.0,221.0 490.0,221.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-3\" stroke-width=\"2px\" d=\"M512,227.0 512,202.0 644.0,202.0 644.0,227.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-3\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">pcomp</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M644.0,229.0 L648.0,221.0 640.0,221.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-4\" stroke-width=\"2px\" d=\"M812,227.0 812,202.0 944.0,202.0 944.0,227.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-4\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">compound</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M812,229.0 L808,221.0 816,221.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-5\" stroke-width=\"2px\" d=\"M662,227.0 662,177.0 947.0,177.0 947.0,227.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-5\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">dobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M947.0,229.0 L951.0,221.0 943.0,221.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-6\" stroke-width=\"2px\" d=\"M662,227.0 662,152.0 1100.0,152.0 1100.0,227.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-6\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">prep</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1100.0,229.0 L1104.0,221.0 1096.0,221.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-7\" stroke-width=\"2px\" d=\"M1262,227.0 1262,177.0 1547.0,177.0 1547.0,227.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-7\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">quantmod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1262,229.0 L1258,221.0 1266,221.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-8\" stroke-width=\"2px\" d=\"M1412,227.0 1412,202.0 1544.0,202.0 1544.0,227.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-8\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">compound</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1412,229.0 L1408,221.0 1416,221.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-9\" stroke-width=\"2px\" d=\"M1112,227.0 1112,152.0 1550.0,152.0 1550.0,227.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-9\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">pobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1550.0,229.0 L1554.0,221.0 1546.0,221.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "displacy.render(doc, style='dep', options=options, jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Visualize Named Entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:24:23.892155Z",
     "start_time": "2018-11-22T18:24:23.885944Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"entities\" style=\"line-height: 2.5\">\n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    Apple\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " is looking at buying \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    U.K.\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " startup for \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    $1 billion\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">MONEY</span>\n",
       "</mark>\n",
       "</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "displacy.render(doc, style='ent', jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Read BBC Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now read a larger set of 2,225 BBC News articles (see GitHub for data source details) that belong to five categories and are stored in individual text files. We \n",
    "- call the .glob() method of the pathlib’s Path object, \n",
    "- iterate over the resulting list of paths, \n",
    "- read all lines of the news article excluding the heading in the first line, and \n",
    "- append the cleaned result to a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:24:44.865932Z",
     "start_time": "2018-11-22T18:24:44.780346Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "files = (DATA_DIR / 'bbc').glob('**/*.txt')\n",
    "bbc_articles = []\n",
    "for i, file in enumerate(files):\n",
    "    with file.open(encoding='latin1') as f:\n",
    "        lines = f.readlines()\n",
    "        body = ' '.join([l.strip() for l in lines[1:]]).strip()\n",
    "        bbc_articles.append(body)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:24:44.870224Z",
     "start_time": "2018-11-22T18:24:44.867824Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2225"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(bbc_articles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:24:44.877781Z",
     "start_time": "2018-11-22T18:24:44.871120Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Voting is under way for the annual Bloggies which recognise the best web blogs - online spaces where people publish their thoughts - of the year.  Nominations were announced on Sunday, but traffic to the official site was so heavy that the website was temporarily closed because of too many visitors. Weblogs have been nominated in 30 categories, from the top regional blog, to the best-kept-secret blog. Blogs had a huge year, with a top US dictionary naming \"blog\" word of 2004. Technorati, a blog search engine, tracks about six million blogs and says that more than 12,000 are added daily. A blog is created every 5.8 seconds, according to US research think-tank Pew Internet and American Life, but less than 40% of the total are updated at least once every two months.  Nikolai Nolan, who has run the Bloggies for the past five years, told the BBC News website he was not too surprised by the amount of voters who crowded the site. \"The awards always get a lot of traffic; this was just my first year on a server with a bandwidth limit, so I had to guess how much I\\'d need,\" he said. There were many new finalists this year, he added, and a few that had won Bloggies before. Several entries reflected specific news events. \"There are four nominations for the South-East Asia Earthquake and Tsunami Blog, which is a pretty timely one for 2005,\" said Mr Nolan.  The big Bloggies battle will be for the ultimate prize of blog of the year. The nominated blogs are wide-ranging covering what is in the news to quirky sites of interest. Fighting it out for the coveted award are Gawker, This Fish Needs a Bicycle, Wonkette, Boing Boing, and Gothamist. In a sign that blogs are playing an increasingly key part in spreading news and current affairs, The South-East Asia Earthquake and Tsunami Blog is also nominated in the best overall category. GreenFairyDotcom, Londonist, Hicksdesign, PlasticBag and London Underground Tube Blog are the nominees in the best British or Irish weblog.  Included in the other categories is best \"meme\". This is for the top \"replicating idea that spread about weblogs\". Nominations include Flickr, a web photo album which lets people upload, tag, share and publish their images to blogs. Podcasting has also made an appearance in the category. It is an increasingly popular idea that makes use of RSS (really simple syndication) and audio technology to let people easily make their own radio shows, and distribute them automatically onto portable devices.  Many are done by those who already have text-based blogs, so they are almost like audio blogs. Three new categories have been added to the list this year, including best food, best entertainment, and best writing of a weblog. One of the categories that was scrapped though was best music blog. The winners of the fifth annual Bloggies are chosen by the public. Public voting closes on 3 February and the winners will be announced sometime between 13 and 15 March.'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bbc_articles[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Parse first article through Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:24:44.885852Z",
     "start_time": "2018-11-22T18:24:44.879325Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['tagger', 'parser', 'ner']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp.pipe_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:24:44.985896Z",
     "start_time": "2018-11-22T18:24:44.887735Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "spacy.tokens.doc.Doc"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc = nlp(bbc_articles[0])\n",
    "type(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Detect sentence boundary\n",
    "Sentence boundaries are calculated from the syntactic parse tree, so features such as punctuation and capitalisation play an important but non-decisive role in determining the sentence boundaries. \n",
    "\n",
    "Usually this means that the sentence boundaries will at least coincide with clause boundaries, even given poorly punctuated text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "spaCy computes sentence boundaries from the syntactic parse tree so that punctuation and capitalization play an important but not decisive role. As a result, boundaries will coincide with clause boundaries, even for poorly punctuated text.\n",
    "\n",
    "We can access the parsed sentences using the .sents attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:24:44.990720Z",
     "start_time": "2018-11-22T18:24:44.987118Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Voting is under way for the annual Bloggies which recognise the best web blogs - online spaces where people publish their thoughts - of the year.  ,\n",
       " Nominations were announced on Sunday, but traffic to the official site was so heavy that the website was temporarily closed because of too many visitors.,\n",
       " Weblogs have been nominated in 30 categories, from the top regional blog, to the best-kept-secret blog.]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences = [s for s in doc.sents]\n",
    "sentences[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:24:45.004845Z",
     "start_time": "2018-11-22T18:24:44.992156Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['as_doc', 'doc', 'end', 'end_char', 'ent_id', 'ent_id_', 'ents', 'get_extension', 'get_lca_matrix', 'has_extension', 'has_vector', 'label', 'label_', 'lefts', 'lemma_', 'lower_', 'merge', 'n_lefts', 'n_rights', 'noun_chunks', 'orth_', 'remove_extension', 'rights', 'root', 'sent', 'sentiment', 'set_extension', 'similarity', 'start', 'start_char', 'string', 'subtree', 'text', 'text_with_ws', 'to_array', 'upper_', 'vector', 'vector_norm', 'vocab'] "
     ]
    }
   ],
   "source": [
    "get_attributes(sentences[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:24:45.024224Z",
     "start_time": "2018-11-22T18:24:45.006282Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Token</th>\n",
       "      <th>POS Tag</th>\n",
       "      <th>Meaning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Voting</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>noun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>is</td>\n",
       "      <td>VERB</td>\n",
       "      <td>verb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>under</td>\n",
       "      <td>ADP</td>\n",
       "      <td>adposition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>way</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>noun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>for</td>\n",
       "      <td>ADP</td>\n",
       "      <td>adposition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>the</td>\n",
       "      <td>DET</td>\n",
       "      <td>determiner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>annual</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>adjective</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Bloggies</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>proper noun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>which</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>adjective</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>recognise</td>\n",
       "      <td>VERB</td>\n",
       "      <td>verb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>the</td>\n",
       "      <td>DET</td>\n",
       "      <td>determiner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>best</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>adjective</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>web</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>noun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>blogs</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>noun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>-</td>\n",
       "      <td>PUNCT</td>\n",
       "      <td>punctuation</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Token POS Tag      Meaning\n",
       "0      Voting    NOUN         noun\n",
       "1          is    VERB         verb\n",
       "2       under     ADP   adposition\n",
       "3         way    NOUN         noun\n",
       "4         for     ADP   adposition\n",
       "5         the     DET   determiner\n",
       "6      annual     ADJ    adjective\n",
       "7    Bloggies   PROPN  proper noun\n",
       "8       which     ADJ    adjective\n",
       "9   recognise    VERB         verb\n",
       "10        the     DET   determiner\n",
       "11       best     ADJ    adjective\n",
       "12        web    NOUN         noun\n",
       "13      blogs    NOUN         noun\n",
       "14          -   PUNCT  punctuation"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame([[t.text, t.pos_, spacy.explain(t.pos_)] for t in sentences[0]], \n",
    "             columns=['Token', 'POS Tag', 'Meaning']).head(15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:24:45.033202Z",
     "start_time": "2018-11-22T18:24:45.025531Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" id=\"0\" class=\"displacy\" width=\"3800\" height=\"662.0\" style=\"max-width: none; height: 662.0px; color: white; background: #09a3d5; font-family: Source Sans Pro\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">Voting</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"200\">is</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"200\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"350\">under</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"350\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"500\">way</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"500\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"650\">for</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"650\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"800\">the</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"800\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"950\">annual</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"950\">ADJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1100\">Bloggies</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1100\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1250\">which</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1250\">ADJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1400\">recognise</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1400\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1550\">the</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1550\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1700\">best</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1700\">ADJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1850\">web</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1850\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2000\">blogs -</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2000\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2150\">online</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2150\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2300\">spaces</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2300\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2450\">where</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2450\">ADV</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2600\">people</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2600\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2750\">publish</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2750\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2900\">their</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2900\">ADJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"3050\">thoughts -</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"3050\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"3200\">of</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"3200\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"3350\">the</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"3350\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"3500\">year.</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"3500\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"572.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"3650\"> </tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"3650\"></tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-0\" stroke-width=\"2px\" d=\"M62,527.0 62,502.0 182.0,502.0 182.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-0\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M62,529.0 L58,521.0 66,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-1\" stroke-width=\"2px\" d=\"M212,527.0 212,502.0 332.0,502.0 332.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-1\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">prep</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M332.0,529.0 L336.0,521.0 328.0,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-2\" stroke-width=\"2px\" d=\"M362,527.0 362,502.0 482.0,502.0 482.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-2\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">pobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M482.0,529.0 L486.0,521.0 478.0,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-3\" stroke-width=\"2px\" d=\"M212,527.0 212,452.0 638.0,452.0 638.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-3\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">prep</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M638.0,529.0 L642.0,521.0 634.0,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-4\" stroke-width=\"2px\" d=\"M812,527.0 812,477.0 1085.0,477.0 1085.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-4\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M812,529.0 L808,521.0 816,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-5\" stroke-width=\"2px\" d=\"M962,527.0 962,502.0 1082.0,502.0 1082.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-5\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">amod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M962,529.0 L958,521.0 966,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-6\" stroke-width=\"2px\" d=\"M662,527.0 662,452.0 1088.0,452.0 1088.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-6\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">pobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1088.0,529.0 L1092.0,521.0 1084.0,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-7\" stroke-width=\"2px\" d=\"M1262,527.0 1262,502.0 1382.0,502.0 1382.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-7\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1262,529.0 L1258,521.0 1266,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-8\" stroke-width=\"2px\" d=\"M1112,527.0 1112,477.0 1385.0,477.0 1385.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-8\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">relcl</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1385.0,529.0 L1389.0,521.0 1381.0,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-9\" stroke-width=\"2px\" d=\"M1562,527.0 1562,402.0 2294.0,402.0 2294.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-9\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1562,529.0 L1558,521.0 1566,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-10\" stroke-width=\"2px\" d=\"M1712,527.0 1712,427.0 2291.0,427.0 2291.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-10\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">amod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1712,529.0 L1708,521.0 1716,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-11\" stroke-width=\"2px\" d=\"M1862,527.0 1862,502.0 1982.0,502.0 1982.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-11\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">compound</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1862,529.0 L1858,521.0 1866,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-12\" stroke-width=\"2px\" d=\"M2012,527.0 2012,502.0 2132.0,502.0 2132.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-12\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">compound</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M2012,529.0 L2008,521.0 2016,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-13\" stroke-width=\"2px\" d=\"M2162,527.0 2162,502.0 2282.0,502.0 2282.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-13\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">amod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M2162,529.0 L2158,521.0 2166,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-14\" stroke-width=\"2px\" d=\"M1412,527.0 1412,377.0 2297.0,377.0 2297.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-14\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">dobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M2297.0,529.0 L2301.0,521.0 2293.0,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-15\" stroke-width=\"2px\" d=\"M2462,527.0 2462,477.0 2735.0,477.0 2735.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-15\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">advmod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M2462,529.0 L2458,521.0 2466,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-16\" stroke-width=\"2px\" d=\"M2612,527.0 2612,502.0 2732.0,502.0 2732.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-16\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M2612,529.0 L2608,521.0 2616,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-17\" stroke-width=\"2px\" d=\"M2312,527.0 2312,452.0 2738.0,452.0 2738.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-17\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">relcl</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M2738.0,529.0 L2742.0,521.0 2734.0,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-18\" stroke-width=\"2px\" d=\"M2912,527.0 2912,502.0 3032.0,502.0 3032.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-18\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">poss</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M2912,529.0 L2908,521.0 2916,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-19\" stroke-width=\"2px\" d=\"M2762,527.0 2762,477.0 3035.0,477.0 3035.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-19\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">dobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M3035.0,529.0 L3039.0,521.0 3031.0,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-20\" stroke-width=\"2px\" d=\"M2762,527.0 2762,452.0 3188.0,452.0 3188.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-20\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">prep</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M3188.0,529.0 L3192.0,521.0 3184.0,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-21\" stroke-width=\"2px\" d=\"M3362,527.0 3362,502.0 3482.0,502.0 3482.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-21\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M3362,529.0 L3358,521.0 3366,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-22\" stroke-width=\"2px\" d=\"M212,527.0 212,352.0 3500.0,352.0 3500.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-22\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">punct</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M3500.0,529.0 L3504.0,521.0 3496.0,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-23\" stroke-width=\"2px\" d=\"M3512,527.0 3512,502.0 3632.0,502.0 3632.0,527.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-23\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\"></textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M3632.0,529.0 L3636.0,521.0 3628.0,521.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "options = {'compact': True, 'bg': '#09a3d5',\n",
    "           'color': 'white', 'font': 'Source Sans Pro'}\n",
    "displacy.render(sentences[0].as_doc(), style='dep', jupyter=True, options=options)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:24:45.040053Z",
     "start_time": "2018-11-22T18:24:45.034511Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "annual | DATE | Absolute or relative dates or periods\n",
      "the | DATE | Absolute or relative dates or periods\n",
      "year | DATE | Absolute or relative dates or periods\n"
     ]
    }
   ],
   "source": [
    "for t in sentences[0]:\n",
    "    if t.ent_type_:\n",
    "        print('{} | {} | {}'.format(t.text, t.ent_type_, spacy.explain(t.ent_type_)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:24:45.051981Z",
     "start_time": "2018-11-22T18:24:45.041838Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"entities\" style=\"line-height: 2.5\">Voting is under way for the \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    annual\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       " Bloggies which recognise the best web blogs - online spaces where people publish their thoughts - of \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    the year\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       ".  </div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "displacy.render(sentences[0].as_doc(), style='ent', jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Named Entity-Recognition with textacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "spaCy enables named entity recognition using the .ent_type_ attribute:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Textacy makes access to the named entities that appear in the first article easy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:24:45.061541Z",
     "start_time": "2018-11-22T18:24:45.053653Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "year                          4\n",
       "US                            2\n",
       "annual                        2\n",
       "Tsunami Blog                  2\n",
       "South-East Asia Earthquake    2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from textacy.extract import named_entities\n",
    "entities = [e.text for e in named_entities(doc)]\n",
    "pd.Series(entities).value_counts().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### N-Grams with textacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "N-grams combine N consecutive tokens. This can be useful for the bag-of-words model because, depending on the textual context, treating, e.g, ‘data scientist’ as a single token may be more meaningful than the two distinct tokens ‘data’ and ‘scientist’.\n",
    "\n",
    "Textacy makes it easy to view the ngrams of a given length n occurring with at least min_freq times:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:24:45.072572Z",
     "start_time": "2018-11-22T18:24:45.062472Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Tsunami Blog       2\n",
       "annual Bloggies    2\n",
       "East Asia          2\n",
       "Asia Earthquake    2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from textacy.extract import ngrams\n",
    "pd.Series([n.text for n in ngrams(doc, n=2, min_freq=2)]).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The spaCy streaming Pipeline API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To pass a larger number of documents through the processing pipeline, we can use spaCy’s streaming API as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:26:32.075601Z",
     "start_time": "2018-11-22T18:24:45.073615Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 "
     ]
    }
   ],
   "source": [
    "iter_texts = (bbc_articles[i] for i in range(len(bbc_articles)))\n",
    "for i, doc in enumerate(nlp.pipe(iter_texts, batch_size=50, n_threads=8)):\n",
    "    if i % 100 == 0:\n",
    "        print(i, end = ' ')\n",
    "    assert doc.is_parsed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Multi-language Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "spaCy includes trained language models for English, German, Spanish, Portuguese, French, Italian and Dutch, as well as a multi-language model for named-entity recognition. Cross-language usage is straightforward since the API does not change.\n",
    "\n",
    "We will illustrate the Spanish language model using a parallel corpus of TED talk subtitles. For this purpose, we instantiate both language models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### Create a Spanish Language Object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:26:32.778571Z",
     "start_time": "2018-11-22T18:26:32.076935Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "model = {}\n",
    "for language in ['en', 'es']:\n",
    "    model[language] = spacy.load(language) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Read bilingual TED2013 samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:29:55.500761Z",
     "start_time": "2018-11-22T18:29:55.490626Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "text = {}\n",
    "path = Path('data/TED')\n",
    "for language in ['en', 'es']:\n",
    "    file_name = path /  'TED2013_sample.{}'.format(language)\n",
    "    text[language] = file_name.read_text()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Sentence Boundaries English vs Spanish"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:29:58.265211Z",
     "start_time": "2018-11-22T18:29:58.161340Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentences: en 19\n",
      "Sentences: es 22\n"
     ]
    }
   ],
   "source": [
    "parsed, sentences = {}, {}\n",
    "for language in ['en', 'es']:\n",
    "    parsed[language] = model[language](text[language])\n",
    "    sentences[language] = list(parsed[language].sents)\n",
    "    print('Sentences:', language, len(sentences[language]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:29:58.321011Z",
     "start_time": "2018-11-22T18:29:58.317422Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " 1\n",
      "English:\t There's a tight and surprising link between the ocean's health and ours, says marine biologist Stephen Palumbi.\n",
      "Spanish:\t Existe una estrecha y sorprendente relación entre nuestra salud y la salud del océano, dice el biologo marino Stephen Palumbi.\n",
      "\n",
      " 2\n",
      "English:\t He shows how toxins at the bottom of the ocean food chain find their way into our bodies, with a shocking story of toxic contamination from a Japanese fish market.\n",
      "Spanish:\t Nos muestra, através de una impactante historia acerca de la contaminación tóxica en el mercado pesquero japonés, como las toxinas de la cadena alimenticia del fondo oceánico llegan a nuestro cuerpo.\n",
      "\n",
      " 3\n",
      "English:\t His work points a way forward for saving the oceans' health -- and humanity's. fish,health,mission blue,oceans,science 899 Stephen Palumbi: Following the mercury trail It can be a very complicated thing, the ocean.\n",
      "Spanish:\t fish,health,mission blue,oceans,science 899 Stephen Palumbi: Siguiendo el camino del mercurio.\n",
      "\n",
      " 4\n",
      "English:\t And it can be a very complicated thing, what human health is.\n",
      "Spanish:\t El océano puede ser una cosa muy complicada.\n",
      "\n",
      " 5\n",
      "English:\t And bringing those two together might seem a very daunting task, but what I'm going to try to say is that even in that complexity, there's some simple themes that I think, if we understand, we can really move forward.\n",
      "Spanish:\t Y podria ser una cosa muy complicada lo que la salud humana es.\n",
      "\n",
      " 6\n",
      "English:\t And those simple themes aren't really themes about the complex science of what's going on, but things that we all pretty well know.\n",
      "Spanish:\t Y unirlas, podría ser una tarea desalentadora.\n"
     ]
    }
   ],
   "source": [
    "for i, (en, es) in enumerate(zip(sentences['en'], sentences['es']), 1):\n",
    "    print('\\n', i)\n",
    "    print('English:\\t', en)\n",
    "    print('Spanish:\\t', es)\n",
    "    if i > 5: \n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### POS Tagging English vs Spanish"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:29:58.650120Z",
     "start_time": "2018-11-22T18:29:58.639202Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "pos = {}\n",
    "for language in ['en', 'es']:\n",
    "    pos[language] = pd.DataFrame([[t.text, t.pos_, spacy.explain(t.pos_)] for t in sentences[language][0]],\n",
    "                                 columns=['Token', 'POS Tag', 'Meaning'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:29:58.822232Z",
     "start_time": "2018-11-22T18:29:58.801912Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Token</th>\n",
       "      <th>POS Tag</th>\n",
       "      <th>Meaning</th>\n",
       "      <th>Token</th>\n",
       "      <th>POS Tag</th>\n",
       "      <th>Meaning</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>There</td>\n",
       "      <td>ADV</td>\n",
       "      <td>adverb</td>\n",
       "      <td>Existe</td>\n",
       "      <td>VERB</td>\n",
       "      <td>verb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>'s</td>\n",
       "      <td>VERB</td>\n",
       "      <td>verb</td>\n",
       "      <td>una</td>\n",
       "      <td>DET</td>\n",
       "      <td>determiner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>a</td>\n",
       "      <td>DET</td>\n",
       "      <td>determiner</td>\n",
       "      <td>estrecha</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>adjective</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>tight</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>adjective</td>\n",
       "      <td>y</td>\n",
       "      <td>CONJ</td>\n",
       "      <td>conjunction</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>and</td>\n",
       "      <td>CCONJ</td>\n",
       "      <td>coordinating conjunction</td>\n",
       "      <td>sorprendente</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>adjective</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>surprising</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>adjective</td>\n",
       "      <td>relación</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>noun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>link</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>noun</td>\n",
       "      <td>entre</td>\n",
       "      <td>ADP</td>\n",
       "      <td>adposition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>between</td>\n",
       "      <td>ADP</td>\n",
       "      <td>adposition</td>\n",
       "      <td>nuestra</td>\n",
       "      <td>DET</td>\n",
       "      <td>determiner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>the</td>\n",
       "      <td>DET</td>\n",
       "      <td>determiner</td>\n",
       "      <td>salud</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>noun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>ocean</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>noun</td>\n",
       "      <td>y</td>\n",
       "      <td>CONJ</td>\n",
       "      <td>conjunction</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>'s</td>\n",
       "      <td>PART</td>\n",
       "      <td>particle</td>\n",
       "      <td>la</td>\n",
       "      <td>DET</td>\n",
       "      <td>determiner</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>health</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>noun</td>\n",
       "      <td>salud</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>noun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>and</td>\n",
       "      <td>CCONJ</td>\n",
       "      <td>coordinating conjunction</td>\n",
       "      <td>del</td>\n",
       "      <td>ADP</td>\n",
       "      <td>adposition</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>ours</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>noun</td>\n",
       "      <td>océano</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>noun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>,</td>\n",
       "      <td>PUNCT</td>\n",
       "      <td>punctuation</td>\n",
       "      <td>,</td>\n",
       "      <td>PUNCT</td>\n",
       "      <td>punctuation</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Token POS Tag                   Meaning         Token POS Tag  \\\n",
       "0        There     ADV                    adverb        Existe    VERB   \n",
       "1           's    VERB                      verb           una     DET   \n",
       "2            a     DET                determiner      estrecha     ADJ   \n",
       "3        tight     ADJ                 adjective             y    CONJ   \n",
       "4          and   CCONJ  coordinating conjunction  sorprendente     ADJ   \n",
       "5   surprising     ADJ                 adjective      relación    NOUN   \n",
       "6         link    NOUN                      noun         entre     ADP   \n",
       "7      between     ADP                adposition       nuestra     DET   \n",
       "8          the     DET                determiner         salud    NOUN   \n",
       "9        ocean    NOUN                      noun             y    CONJ   \n",
       "10          's    PART                  particle            la     DET   \n",
       "11      health    NOUN                      noun         salud    NOUN   \n",
       "12         and   CCONJ  coordinating conjunction           del     ADP   \n",
       "13        ours    NOUN                      noun        océano    NOUN   \n",
       "14           ,   PUNCT               punctuation             ,   PUNCT   \n",
       "\n",
       "        Meaning  \n",
       "0          verb  \n",
       "1    determiner  \n",
       "2     adjective  \n",
       "3   conjunction  \n",
       "4     adjective  \n",
       "5          noun  \n",
       "6    adposition  \n",
       "7    determiner  \n",
       "8          noun  \n",
       "9   conjunction  \n",
       "10   determiner  \n",
       "11         noun  \n",
       "12   adposition  \n",
       "13         noun  \n",
       "14  punctuation  "
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bilingual_parsed = pd.concat([pos['en'], pos['es']], axis=1)\n",
    "bilingual_parsed.head(5).to_csv('data/bilingual.csv', index=False)\n",
    "bilingual_parsed.head(15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:29:59.172621Z",
     "start_time": "2018-11-22T18:29:59.168637Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" id=\"0\" class=\"displacy\" width=\"3050\" height=\"437.0\" style=\"max-width: none; height: 437.0px; color: white; background: #09a3d5; font-family: Source Sans Pro\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">Existe</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"200\">una</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"200\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"350\">estrecha</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"350\">ADJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"500\">y</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"500\">CONJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"650\">sorprendente</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"650\">ADJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"800\">relación</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"800\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"950\">entre</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"950\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1100\">nuestra</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1100\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1250\">salud</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1250\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1400\">y</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1400\">CONJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1550\">la</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1550\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1700\">salud</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1700\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1850\">del</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1850\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2000\">océano,</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2000\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2150\">dice</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2150\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2300\">el</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2300\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2450\">biologo</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2450\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2600\">marino</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2600\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2750\">Stephen</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2750\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"347.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2900\">Palumbi.</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2900\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-0\" stroke-width=\"2px\" d=\"M62,302.0 62,202.0 2150.0,202.0 2150.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-0\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">ccomp</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M62,304.0 L58,296.0 66,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-1\" stroke-width=\"2px\" d=\"M212,302.0 212,277.0 341.0,277.0 341.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-1\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M212,304.0 L208,296.0 216,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-2\" stroke-width=\"2px\" d=\"M62,302.0 62,252.0 344.0,252.0 344.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-2\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M344.0,304.0 L348.0,296.0 340.0,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-3\" stroke-width=\"2px\" d=\"M512,302.0 512,252.0 794.0,252.0 794.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-3\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">cc</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M512,304.0 L508,296.0 516,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-4\" stroke-width=\"2px\" d=\"M662,302.0 662,277.0 791.0,277.0 791.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-4\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">amod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M662,304.0 L658,296.0 666,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-5\" stroke-width=\"2px\" d=\"M362,302.0 362,227.0 797.0,227.0 797.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-5\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">conj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M797.0,304.0 L801.0,296.0 793.0,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-6\" stroke-width=\"2px\" d=\"M962,302.0 962,252.0 1244.0,252.0 1244.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-6\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">case</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M962,304.0 L958,296.0 966,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-7\" stroke-width=\"2px\" d=\"M1112,302.0 1112,277.0 1241.0,277.0 1241.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-7\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1112,304.0 L1108,296.0 1116,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-8\" stroke-width=\"2px\" d=\"M812,302.0 812,227.0 1247.0,227.0 1247.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-8\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">nmod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1247.0,304.0 L1251.0,296.0 1243.0,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-9\" stroke-width=\"2px\" d=\"M1412,302.0 1412,252.0 1694.0,252.0 1694.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-9\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">cc</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1412,304.0 L1408,296.0 1416,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-10\" stroke-width=\"2px\" d=\"M1562,302.0 1562,277.0 1691.0,277.0 1691.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-10\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1562,304.0 L1558,296.0 1566,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-11\" stroke-width=\"2px\" d=\"M1262,302.0 1262,227.0 1697.0,227.0 1697.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-11\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">conj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1697.0,304.0 L1701.0,296.0 1693.0,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-12\" stroke-width=\"2px\" d=\"M1862,302.0 1862,277.0 1991.0,277.0 1991.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-12\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">case</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1862,304.0 L1858,296.0 1866,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-13\" stroke-width=\"2px\" d=\"M1712,302.0 1712,252.0 1994.0,252.0 1994.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-13\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">nmod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1994.0,304.0 L1998.0,296.0 1990.0,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-14\" stroke-width=\"2px\" d=\"M2312,302.0 2312,277.0 2441.0,277.0 2441.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-14\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M2312,304.0 L2308,296.0 2316,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-15\" stroke-width=\"2px\" d=\"M2162,302.0 2162,252.0 2444.0,252.0 2444.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-15\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M2444.0,304.0 L2448.0,296.0 2440.0,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-16\" stroke-width=\"2px\" d=\"M2462,302.0 2462,277.0 2591.0,277.0 2591.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-16\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">amod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M2591.0,304.0 L2595.0,296.0 2587.0,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-17\" stroke-width=\"2px\" d=\"M2462,302.0 2462,252.0 2744.0,252.0 2744.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-17\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">appos</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M2744.0,304.0 L2748.0,296.0 2740.0,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-0-18\" stroke-width=\"2px\" d=\"M2762,302.0 2762,277.0 2891.0,277.0 2891.0,302.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-0-18\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">flat</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M2891.0,304.0 L2895.0,296.0 2887.0,296.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "displacy.render(sentences['es'][0].as_doc(), style='dep', jupyter=True, options=options)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "355px",
    "left": "1936.95px",
    "top": "146.953px",
    "width": "305px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
